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IMPLICATIONS 


Foreign  nationals  attend  United  States  Air  Force  pilot  training  schools 
under  the  Mutual  Defense  Assistance  Program  (MDAP)*  The  MDAP  English  Profi¬ 
ciency  Examination  is  currently  being  used  overseas  to  select  personnel  who 
understand  English  well  enough  to  profit  from  this  instruction*  This  Re¬ 
search  Report  describes  the  development  and  the  validation  of  the  first  form 
of  this  test# 

An  experimental  version  of  the  test  was  given  to  foreign  students 
already  in  training  in  this  country#  Analysis  of  these  data  led  to  the  se¬ 
lection  of  90  written  items  (Part  I)  and  24  oral  items  (Part  II)  for  the 
first  operational  version  of  the  test# 

Form  A  was  administered  overseas  in  1953  to  select  personnel  to  attend 
pilot  training  schools  in  the  United  States#  Another  sample  was  tested  ex¬ 
perimentally  at  Lackland  Air  Force  Base#  In  addition  the  test  was  given  to 
basic  airmen#  Analysis  of  these  data  gave  evidence  of  acceptable  reliability 
of  the  test part  scores  as  well  as  its  discriminating  powers# 

An  individually-administered  behavior  test  of  English  comprehension  was 
developed  as  a  criterion  of  English  proficiency,  and  administered  along  with 
MDAP  English  Proficiency  Examination  Form  A.  High  correlation  of  the  English 
Proficiency  Examination  with  the  behavior  test  and  with  an  English  test  used 
in  Pre-Flight  training  gave  assurance  that  it  actually  measures  language 
comprehension#  Predictive  validity  was  found  for  Form  A  through  correlating 
the  test  scores  with  success  in  Pre-Flight  training# 

Use  of  MDAP  English  Proficiency  Examination  Form  A  in  selecting  foreign 
nationals  for  pilot  training  in  the  United  States  can  increase  efficiency  of 
training  through  entering  into  Pre-Flight  only  those  candidates  with  enough 
English  comprehension  to  have  a  reasonable  chance  to  profit  from  the  in¬ 
struction#  To  accomplish  this  goal,  a  sufficiently  large  number  of  applicants 
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must  be  available  so  that  a  suitable  qualifying  level  can  be  set  in  the 
selection  test. 

Since  this  report  was  prepared.  Forms  B  and  C  of  MDAP  English  Proficiency 
Examination  have  been  developed. 
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THE  DEVELOPMENT  AND  VALIDATION 
OF  MDAP  ENGLISH  PROFICIENCY  EXAMINATION  FORM  A 


BACKGROUND1 

This  report  concerns  the  development  and  validation  of  a  test  of 
English  comprehension,.  The  test  is  Intended  for  use  with  foreign  nation¬ 
als  who  are  scheduled  to  attend  United  States  Air  Force  pilot  training 
schools. 

In  August  1952,  the  Air  Force  Personnel  and  Training  Research  Center 
was  directed  by  the  Human  Factors  Division,  Directorate  of  Research  and 
Development,  Headquarters  United  States  Air  Force,  to  develop  an  English 
proficiency  examination  for  selecting  Mutual  Defense  Assistance  Program 
(MDAP)  personnel  to  attend  pilot  training  schools  in  the  United  States 0 
Information  concerning  the  problems  involved  vhs  obtained  from  confer¬ 
ences  with  operating  personnel  at  Goodfellow  Air  Force  Base. 


THE  EXPERIMENTAL  TEST 

Based  upon  facts  learned  in  the  above-mentioned  conferences,  the 
decision  was  made  to  use  both  written  and  oral  materials 0  Items  were  con¬ 
structed  in  areas  of  the  Aircrew  Classification  Battery®  An  attempt  was 
made  to  keep  the  subject  content  of  items  simple  enough  to  insure  that 
proper  answering  of  the  question  would  be  a  function  of  English  proficiency® 

Items  were  constructed  in  the  areas  of  vocabulary,  mechanical  prin¬ 
ciples,  word  order,  interpretation  of  data,  arithmetic  reasoning,  aviation 
information,  reading  comprehension,  and  background  for  current  affairs.  The 
number  of  items  included  and  their  descriptions  are  presented  in  Table  lo 

The  experimental  form  of  the  test,  entitled  MDAP  Pilot  Selection  Test, 
was  a  mimeographed  booklet  containing  97  written  and  24  oral  items  ®  This 
form  was  administered  to  foreign  student  pilots  at  USAF  bases  during  August 
and  September  1952  by  a  traveling  team  of  tpst  administrators® 


Analysis  of  the  Experimental  Test 

The  analysis  of  the  first  experimental  administration  was  restricted 
to  data  obtained  from  students  of  all  countries  in  Class  53-F  and  French 
students  in  Class  53-C0  The  French  students  were  chosen  because  they  were 
the  largest  group  of  students  from  any  one  country  and  provided  large 
samples  first ®  Furthermore,  the  instructors  at  Goodfellow  Air  Force  Base 
reported  that  the  French  students  varied  more  in  their  English  proficiency 
than  did  the  students  from  any  other  country®  The  written  (Part  I)  and 
oral  (Part  II)  portions  of  the  test  were  analysed  separately® 

Manuscript  received  19  April  1934® 
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Table  1 


Types  of  Items  Used  in  English  Proficiency  Testing 

Noc  of  items 
Expsri-  Opera- 


Item  type 

mental 

form 

tior.sl 

form 

Description 

Part  I  (written) 
Vocabulary 

23 

22 

A  sentence  with  one  word  underlined*  Choose 
the  answer  whose  meaning  is  nearest  that  of 
the  underlined  word.  Five  choices. 

Mechanical 

principles 

10 

8 

Situational  mechanical  problems  designed 
to  measure  understanding  of  mechanical 
forces,  movements,  or  principles.  Two  to 
five  choices. 

Word  order 

15 

15 

A  sentence  is  presented  in  three  different 
word  arrargements,  Choose  the  one  which 
states  the  sentence  most  simply 

Interpret  at ion 
of  data 

12 

12 

Simple  bar  graphs  are  presented.  Each 
question  asks  for  a  conclusion  based  on 
the  bar  graph.  Five  choices* 

Arithmetic 

reasoning 

10 

9 

These  are  stated  arithmetic  problems  * 

Five  choices. 

Aviation 

information 

3 

3 

The  stem  asks  for  a  bit  of  information  such 
as  the  function  of  art  airplane  part  or  about, 
a  maneuver*  Five  choices 

Reading 

comprehension 

9 

9 

A  paragraph  is  presented  and  followed  by 
questions  concerning  the  facts  in  the 
paragraph.  Two  to  five  choices 

Background  for  15 

current  affairs 

12 

These  questions  ere  of  historical  or 
geographical  fact.  Five  choices. 

Total  Part.  I 
(written) 

97 

90 

Part  II  (oral) 

2A 

2L 

Three  paragraphs  covering  flying  safety  topics 
are  read  aloud.  After  each  paragraph  is  read, 

the  examinee  has  two  minutes  to  answer  ques¬ 
tions  about  the  paragraph.  These  questions  are 
printed  in  the  test  booklet..  The  same  para¬ 
graph  is  then  read  aloud  again  ard  one  more 
minute  allowed  for  answering  questions  before 
going  on  to  the  next  paragraph-  When  possible 
a  recording  is  used  for  the  oral  part-.  Three 
to  five  choice?. 
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Analysis  of  Test  Scores 


Distribution  statistics  for  Class  53-F  were  computed  for  each  country* 
Using  the  French  cases,  correlations  between  the  two  part-scores  were  com¬ 
puted  separately  for  Classes  53 -F  and  53-Co  These  data  are  presented  in 
Table  2. 


Table  2 

Distribution  Statistics  for  the  Experimental 
Test  by  Countries 


Part  I  Part  II 


Country 

Class 

N 

M 

SD 

M 

SD 

H) 

France 

53-C 

93 

75.4 

6.7 

17.9 

2.3 

.34 

France 

53-F 

96 

58.6 

13.1 

11.3 

4.2 

.51 

Belgium 

53-F 

11 

76.9 

5.1 

17.2 

1.6 

Italy 

53-F 

26 

66*9 

7.8 

14.6 

4.4 

Holland 

53-F 

20 

75.7 

7.9 

17.9 

2.6 

Denmark 

53-F 

29 

77.5 

8.6 

17.8 

2.9 

Not e0 --Data  on  Part  I  are  based  on  all  97  items  of  the  experimental 
version*  Part  I  is  written;  Part  II  is  oral. 


The  report  by  instructors  at  Good fellow  Air  Force  Base  that  French 
students  varied  more  in  their  ability  to  understand  English  than  did 
students  from  other  countries  is  confirmed  by  the  distribution  statistics 
in  Table  2.  The  French  of  Class  53-F  had  the  largest  standard  deviation  on 
Part  I  (written)  and  only  the  Italians  had  a  larger  one  on  Part  II  (oral). 

The  moderate  relationship  between  the  two-part-scores  should  be  notedo  The 
difference  in  the  interpart  correlations  of  o51  and  .34  is  apparently  due 
to  the  restriction  in  range  of  scores  among  the  members  of  Class  53-Co  Also 
to  be  noted  is  the  higher  mean  score  for  the  longer-trained  group o 

Comparison  of  the  scores  of  Class  53-F  students  from  various  countries 
shows  that  the  French  students  generally  were  lower  on  both  parts  of  the 
test  than  were  students  from  the  other  countries.  These  differences  are 
significant  beyond  the  .01  level.  The  students  from  Belgium,  Holland,  and 
Denmark  performed  equally  well  on  the  test.  The  French  students  of  Class 
53-C  had  mean  scores  on  both  parts  equal  to  the  mean  scores  of  the  students 
from  Belgium,  Holland,  and  Denmark  in  Class  53-F. 

Three  reasons  for  the  differences  between  the  Frenchmen  of  Classes 
53-C  and  53-F  present  themselves:  (a)  the  French  students  learn  some  English 
after  they  arrive  in  the  United  States;  (b)  students  weak  in  English  are 
eliminated  from  training;  (c)  the  French  students  of  Class  53-C  may  have 
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been  more  proficient  upon  beginning  flying  training  than  were  those  of 
Class  53-F* 


Answer  sheets  were  available  for  two  classes  of  French  students;  Class 
53-C  had  been  in  training  four  and  one-half  months,  and  Class  53-F  had  been 
in  training  two  weeks 0  The  answer  sheets  were  scored  mright  answers  only*” 
Using  answer  sheets  from  Class  53-U  as  the  upper  group  and  from  Class  53-F 
as  the  lower  group,  phi  coefficients  were  computed  for  each  item  in  the 
test*  This  procedure  gives  an  indication  of  item  validity  if  the  hypothesis 
is  accepted  that  English  proficiency  will  improve  during  training* 


A  second  item  analysis  was  made,  using  data  from  Class  53-F  only*.  In 
this  case  the  answer  sheets  were  split  equally  into  upper  and  lower  groups  on 
the  basis  of  the  scores  for  Parts  I  (written)  and  II  (oral)*  Phi  coeffi¬ 
cients  and  difficulty  levels  (percentage  passing  the  item)  were  computed  for 
each  test  item0  As  a  result  of  these  analyses,  7  of  the  97  written  items 
were  omitted  from  the  operational  test* 


Reliability  Estimate 

To  provide  an  estimate  of  the  reliability  of  the  part-scores  of  the  ex¬ 
perimental  test,  Kuder-Richardson  Formula  21  (3)  was  applied  to  the  data  from 
French  students  of  Class  53-F*  For  Part  I  (written)  this  estimate  was  *89 
and  for  Part  II  (oral)  *71*  These  estimates  were  based  on  the  90  written 
items  and  24  oral  items  selected  for  the  operational  form  of  the  testo 

The  analysis  described  above  completed  the  study  of  the  experimental 
form  ©f  the  test,  the  MDAP  Pilot  Selection  Test*  The  urgent  need  for  an 
operational  test  did  not  allow  a  validation  study  t©  be  performed*  However, 
the  characteristics  of  the  distribution  statistics,  part-score  correlation  co¬ 
efficients,  and  reliability  estimates  indicated  that  the  measuring  instrument 
held  potential  as  a  selection  device* 


THE  OPERATIONAL  TEST 

The  operational  form  of  the  test,  MDAP  English  Proficiency  Examination 
(EPE)  Form  A,  is  composed  of  items  from  the  experimental  form  except  for  seven 
of  the  written  items  which  were  omitted*  Table  1  presents  the  number  of  items 
of  each  subject-matter  category  used  in  the  operational  test*  Copy  for  the 
test  booklet  and  manual  for  administration,  along  with  scoring  keys  for  Parts 
I  (written)  and  II  (oral),  were  submitted  to  higher  headquarters  for  publication* 
Sound  tracks  were  prepared  on  16  mm*  film  for  Part  II  (oral)  of  the  test  by 
the  Audio-Yisual  Research  Division  of  the  Human  Factors  Operations  Research 
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Laboratories©  These  sound  tracks  insure  standard  administration  of  the 
oral  part  of  the  test© 


Operational  Administration 

During  the  spring  of  1953 $  MDAP  EPE  Form  A  was  administered  operation¬ 
ally  in  a  number  of  European  countries  under  the  supervision  of  the  Military 
Assistance  Advisory  Groups©  Because  the  16  mm©  sound  tracks  were  not  avail¬ 
able  for  use  in  this  administration,  the  oral  paragraphs  were  read  aloud  from 
the  manual  by  test  administrators©  The  results  of  this  testing  were  used  to 
select  military  personnel  from  various  participating  countries  to  attend 
pilot  training  schools  in  the  United  States© 


Analysis  of  the  Operational  Administration 


Item  Analysis 

Upon  receipt  of  answer  sheets  from  the  operational  administration  of 
MDAP  EPE  Form  A,  the  items  of  each  part  of  the  test  were  analyzed©  The 
sample  contained  523  cases  from  six  countries0  The  analysis  of  each  part 
was  based  on  the  high  and  low  21%  of  the  answer  sheets  for  that  part©  Phi 
coefficients  and  difficulty  levels  were  computed  for  each  testo  A  summary 
of  these  analyses  is  presented  in  Table  3© 

The  figures  in  Table  3  show  that  certain  types  of  items  are  easier  than 
others  for  the  population  tested©  Word-order  items  and  interpretation-of- 
data  items  were  among  the  easiest,  while  the  aviation  information  items  were 
the  most  difficult  for  this  sample  of  potential  foreign  student  pilots© 

The  mean  phi  coefficients  in  Table  3  indicate  the  relationship  of  each 
group  of  items  to  its  appropriate  score,  i©e«,  written  or  oral©  They  are 
indexes  of  internal  consistency.  The  reading  comprehension  items  have  the 
highest  index  of  internal  consistency;  that  is,  they  are  the  most  representa¬ 
tive  of  the  written  part  as  a  whole.  Mechanical  principles,  aviation  in¬ 
formation,  and  background  for  current  affairs,  as  groups  of  items,  are  less 
related  to  the  total  written  scores© 


Analysis  of  Test  Scores 

Means  and  standard  deviations  of  the  score  distributions  from  each 
country  were  computed©  These  distribution  statistics  are  presented  in 
Table  4° 

The  sample  used  for  the  computations  in  Table  3  was  larger  than  the 
sample  for  the  data  in  Table  4  because  additional  cases  had  been  received 
from  overseas.  The  figures  in  Table  4  show  the  variation  in  English  pro¬ 
ficiency  scores  among  the  potential  student  pilots  by  country©  The  French 
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Table  3 


Summary  of  the  Item  Analysis  of  MDAP  English  Proficiency  Examination 

Form  A£  Overseas  Administration 

(Sample;  523  foreign  student  pilots;  testing  dates;  March-April  1953) 


Items 

Item  type _ 

Part  I  (written) 

1-22 

Vocabulary 

23-30 

Mechanical  principles 

31-45 

Word  order 

46-57 

Interpretation  of  data 

58-66 

Arithmetic  reasoning 

6? -69 

Aviation  information 

70-78 

Reading  comprehension 

79-90 

Background  for  current- 
affairs 

91-114 

Part  IT  (oral) 

Mean 

Moan 

difficulty 

phi 

level 

coefficient 

<55 

<39 

<68 

<23 

<84 

<34 

<81 

<36 

<76 

<33 

<40 

<27 

.61 

<52 

<63 

0^ 

cv 

o 

<57 

c  ZfA 

Table  4 

Part.-Score  Distribution  Statistics 
Prom  the  Operational  Administration 
Of  MDAP  English  Proficiency  Examination  Form  A 


Part  I 
(written) 


Part  II 
( oral ) 


Country 

N 

M 

SD 

M 

SD 

Belgium 

68 

64<1 

10<9 

16<0 

3<9 

Denms  rk 

32 

79  =  6 

6-8 

19<7 

2.4 

Holland 

18 

66<  4 

10<  7 

16<8 

3<8 

France 

320 

61-2 

10<8 

13.4 

4-5 

Portugal 

12 

64  <9 

3<5 

10<  8 

3-8 

Turkey 

4  A 

66<  2 

4  <8 

13  <6 

I<9 

Total 

49A 

63  <4 

11<2 

14<2 

4.5 
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have  a  wider  range  of  talent  than  some  of  the  other  countries®  They  also 
represent  the  largest  group  tested  in  any  of  the  countries® 

The  correlation  between  the  two  part- scores  of  the  MDAP  EPE  Form  A 
was  computed  for  523  cases  collected  under  operational  conditions  overseas® 
The  written  vs®  oral  correlations  was  <>56  as  compared  with  that  of  .51  ob¬ 
tained  from  the  experimental  form  on  the  French  students  of  Class  53-F® 


The  United  States  Administration 

From  February  1953  through  December  1954»  MDAP  EPE  Form  A  was  ad¬ 
ministered  experimentally  to  all  foreign  students  in  Pre-Flight  training 
at  Lackland  Air  Force  Base.  This  testing  was  done  by  highly  trained  test 
administrators,  and  the  16  mm®  sound  track  was  used  to  present  the  oral 
part  of  the  test® 


Item  Analysis 

An  item  analysis  was  made  on  a  sample  of  503  cases  tested  at  Lackland 
Air  Force  Base  in  Pre-Flight  Classes  53-1  through  53-0.  Some  of  the  later 
cases  had  already  been  selected  to  attend  pilot  training  school  on  the 
basis  of  their  operational  test  scores.  Part  I  (written)  and  Part  II  (oral) 
were  analyzed  separately.  The  answer  sheets  of  the  upper  and  lower  21% 
groups  were  selected  on  the  basis  of  the  appropriate  part-score,  and  phi 
coefficients  and  difficulty  levels  were  computed  for  each  test  item®  A 
summary  of  this  analysis  is  presented  in  Table  5« 

Comparison  of  the  mean  difficulty  level  of  each  group  of  items  in 
Tables  3  and  5  shows  that  the  Lackland  administration  reflects  selection 
on  the  basis  of  the  overseas  administration.  The  mean  phi  coefficients 
of  the  item  types  also  show  some  variation  from  overseas  testing  to  the 
Lackland  testing.  Most  of  this  variation  is  associated  with  changes  in 
item  difficulties.  On  the  whole,  the  analysis  shows  the  groups  of  items 
to  be  acceptable  from  the  point  of  view  of  internal  consistency. 


Reliability  Estimate 

Retest  correlations  and  distribution  statistics  were  obtained  for  120 
men  for  whom  both  operational  and  retest  scores  were  available.  These  fig¬ 
ures  are  presented  in  Table  6. 

The  correlation  coefficients  of  .75  and  .73  probably  underestimate  the 
true  reliabilities  of  the  part-scores.  More  than  one  month  elapsed  between 
the  testing  dates.  The  overseas  administration  occurred  at  the  end  of  a 
course  in  English  while  the  Lackland  administration  followed  an  ocean  voyage 
and  introduction  to  the  United  States  Air  Force. 
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Table  5 


Summary  of  Item  Analysis  of  MDAF  English  Proficiency  Examination 
Form  A,,  Lackland  Administration 

(Samples  503  foreign  Pre-Flight  students) 


Item 

Item  type 

Mean 

difficulty 

Mean 

jehi 

1-22 

Part  1  (written) 

Vocabulary 

,68 

.35 

23-30 

Mechanical  principles 

.74 

<19 

31-45 

Word  order 

.90 

.23 

46-57 

Interpretation  of  data 

.92 

el9 

58-66 

Arithmetic  reasoning 

.85 

<27 

67-69 

Aviation  information 

.53 

<37 

70-78 

Reading  comprehension 

.74 

<38 

79-90 

Background  for  current  affairs 

.69 

o28 

90-114 

Part  II  (oral) 

,68 

c50 

Table  6 

Test-Retest  Distribution  Statistics  and  Correlation  Coefficients 
For  MDAP  English  Proficiency  Examination  Form  A 

(Sample;  120  foreign  Pre-Flight  students) 


Part  I  Part  II 


Place  of 
i£§ting__ 

(written) 

M  SD 

(oral) 

M  SD 

Overseas 

73.8 

7.0 

9.6 

3d 

Lackland 

68,9 

8.6 

8-7 

4.1 

rtt  r 

.75 

rtt  * 

,73 

The  lower  part-score  means  and  larger  standard  deviations  of  data 
resulting  from  the  lackland  administration  are  probably  the  result  of 
natural  statistical  phenomena.  It  has  been  found  that  in  any  retest 
situation  a  given  man’s  second  score  is  apt  to  be  nearer  the  average  score 
of  the  group  than  was  his  first  scorer  In  Table  4  it  is  noted  that  the  Part  1 


8 


(written)  mean  of  the  total  overseas  group  was  63o4o  Men  who  made  scores  of 
54  and  below  on  Part  I  (written)  were  not  shipped  to  America,  accounting  for 
the  mean  score  of  73«8  for  the  overseas  testing  of  men  in  Pre-Flight ©  How¬ 
ever,  when  th§s®  men  were  retested  their  mean  scores  moved  closer  to  the 
original  group  means 0 

A  factor  in  the  wider  deviation  of  the  Lackland  testing  is  the  error 
implicit  in  any  measurement — a  number  of  the  men  who  had  achieved  a  score 
of  55  overseas  would  make  lower  scores  on  their  Lackland  retesting*,  thereby 
increasing  the  range  and  standard  deviation  of  the  Lackland  testing o 

The  results  are  complicated  by  unpredictable  changes  in  motivation© 

While  overseas,  the  men  knew  that  failure  on  the  test  would  preclude  their 
coming  to  America©  When  tested  here  they  had  no  information  as  to  the  use 
of  the  test  results© 


Comparison  of  English  Proficiency  Scores 
From  USAF  Basic  Airmen  and  MDAP  Personnel 

The  MDAP  EPE  Form  A  was  administered  to  307  basic  airmen©  The  test 
was  administered  to  the  airmen  exactly  as  it  had  been  to  foreign  Pre-Flight 
students.  The  graph  in  Figure  1  presents  distributions  of  the  Part  I 
(written)  scores  for  basic  airmen,  for  MDAP  students  who  took  the  test  at 
Lackland  Air  Force  Base,  and  for  foreign  personnel  who  were  tested  overseas. 

All  three  distributions  in  Figure  1  demonstrate  a  statistical  charac¬ 
ter  called  ^negative  skewness ."  This  term  is  applied  to  distributions  in 
which  the  more  able  examinees  tend  to  pile  up  on  high  scores  while  the 
scores  of  less  able  persons  are  more  widely  spread  over  the  lower  part  of 
the  score  range.  This  is  a  desirable  distribution  because  it  permits  of 
finer  and  more  accurate  discriminations  among  the  less  able  examinees.  In 
general,  the  curve  representing  the  scores  of  basic  airmen  fits  rather 
closely  the  curve  representing  the  groups  of  MDAP  Pre-Flight  students©  The 
curve  representing  the  scores  obtained  under  operational  conditions  is  more 
nearly  Mnormalm  than  either  of  the  other  two  curves 0  The  shift  to  the  right 
from  the  operational  administration  distribution  to  that  of  the  Lackland 
administration  illustrates  the  selection  that  takes  place  when  the  opera¬ 
tional  test  scores  are  used  to  eliminate  foreign  students  from  participa¬ 
tion  in  the  training  program. 

The  graph  in  Figure  2  compares  the  Part  II  (oral)  score  distributions 
of  the  three  samples  described  above.  The  Part  II  (oral)  score  distribu¬ 
tions  also  show  negative  skewness.  However,  on  the  oral  part  of  the  test 
the  basic  airmen  were  definitely  superior  to  either  of  the  foreign  student 
samples.  The  result  of  using  the  scores  as  a  selection  device  again  is 
evident  in  the  shift  to  the  right  from  the  overseas  administration  to  the 
Lackland  administration. 
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PER  CENT  OF  PERSONS 


RAW  SCORES 


Fir-'.  1.  Distributions  of  Part  I  (written)  scores,  English  Proficiency 
Examination,  for  basic  airmen,  MDAP  students  tested  at  Lackland  Air 
Force,  .base,  and  foreign  personnel  tested  overseas. 
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Fig.  2.  Distributions  of  Part  II  (oral)  scores,  English  Proficiency 
Examination,  for  basic  airmen,  MDAP  students  tested  at  Lackland  Air 
Force  Base,  and  foreign  personnel  tested  overseas. 
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ESTIMATES  OF  VALIDITY 


The  preceding  sections  have  described  the  techniques  used  in  construc¬ 
ting  MDAP  EPE  Form  A0  Various  work  has  been  reported  which  evaluates  the 
test  in  terms  of  reliability  estimates  and  distribution  statistics  for  sev¬ 
eral  samples*  The  present  portion  of  the  report  is  concerned  with  validity* 

Validity  of  two  types  has  been  investigated*  The  first  is  construct 
validity  (2).  The  concept  involves  the  rationale  that  a  test  designed  to 
measure  a  given  dimension  should  correlate  sufficiently  with  other  measures 
of  the  same  dimension*  Thus  by  constructing  two  or  more  instruments  to 
measure  a  specific  dimension  (in  this  case  English  comprehension)  and  cor¬ 
relating  measurements  taken  with  the  instrument s,  validity  can  be  inferred. 

The  second  type  of  validity  under  investigation  is  predictive  validity 
(2).  The  concept  is  one  of  practical  use*  If  test  scores  can  be  found 
predictive  of  a  practical  criterion  (such  as  school  grades) ,  then  the  test 
scores  can  be  used  to  select  those  potential  students  who  are  most  likely 
to  succeed  once  they  have  entered  the  school* 


The  Criteria 


Two  measures  of  English  comprehension,  both  developed  independently  of 
MDAP  EPE  Form  A,  have  been  used  to  gather  evidence  of  construct  validity* 
The  first'  is  a  behavior  test  developed  in  Personnel  Research  Laboratory, 

Air  Force  Personnel  and  Training  Research  Center.  The  second  is  a  test 
developed  by  instructors,  in  Pre-Flight. 


Behavior  Test 

As  an  estimate  of  English  comprehension  independent  of  the  MDAP  EPE 
Form  A,  a  behavior  test  of  28  items  was  developed.  The  test  items  are 
directions  presented  orally  to  the  examinee  in  an  interview  type  of  situ¬ 
ation.  The  directions,  in  simple  English,  are  short  sentences  that  tell 
the  examinee  to  do  something.  If  the  examinee  responds  appropriately,  he 
receives  credit  for  the  item.  The  score  is  the  number  of  correct  responses 
soSide . 


The  equipment  used  in  the  behavior  test  includes  a  pencil,  a  tablet, 
a  book,  a  box  containing  ten  3”  x' 5®*  cards  (one  red  and  nine  white),  a  line 
drawing  of  an  aircraft,  and  a  box,  illustrated  in  Figure  3*  A  table  and 
two  chairs  furnish  the  interview  room.  The  28  behavior  test  items  are  pre¬ 
sented  belows 

*®1*  Show  me  something  you  have  in  your  pockets.  Anything  will  do. 

2.  Put  your  left  hand  on  the  table* 

3.  Walk  around  y@ur  chair  and  sit  down. 


Fig.  3.  Apparatus  for  the  last  ten  items  of  the  behavior  test. 
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4* 

5. 

6. 
7o 
8. 

9. 

10. 
11. 
12. 
13. 
14* 
15  o 
16. 

17. 

18. 

19. 

20. 
21. 
22. 
23  o 
24. 

25  o 
26. 

27. 

28. 


Pall  the  book  toward  you. 

On  th«  picture  of  the  plane  show  me  the  rudder o 
The  propeller. 

The  elevator. 

The  wheels o 
The  Eiotor0 
The  ailerons o 

Using  your  hand,  show  me  how  a  plane  banks,  dives,  and  lands. 
Stand  and  lift  your  chair.  (Now  put  it  down.) 

Writ®  your  name  on  the  tablet 0 

Tear  off  that  page  you  wrote  on  and  put  it  under  the  book. 
Open  the  box.  Find  the  red  card  and  drop  it  on  the  table. 
Push  the  book  to  na« 

Place  the  pencil  in  the  box. 

Slid®  your  feet  back  and  forth  on  the  floor. 

Open  the  door. 

Pull  the  left  lever. 

Push  the  center  button. 

Turn  the  crank © 

Set  the  dial  needle  on  two. 

Throw  the  switch. 

Turn  the  wheel  three  turns. 

Pull  both  end  knobs  up. 

Put  a  ring  on  the  2d  and  4th  pirtSo 
S3.ide  the  button  to  position  two  o'* 


In  administering  tho  behavior  test  the  examinees  were  told  collec¬ 
tively  that  they  ware  to  be  interviewed  to  determine  their  comprehension 
of  English.  The  examinees  were  asked  to  follow  directions  given  by  the 
interviewer.  .  All  testing  was  done  at  the  beginning  of  Pre-Flight  trainings 

Arrangements  were  mad®  to  administer  four  behavior  tests  at  a  time. 
When  possible,  each  administrator  tested  in  a  separate •  roan.  For  some  of 
the  cases,  two  administrators  operated  in  a  large  room,  one  at  each  endo 
Observation  showed  no  effect  on  examinees  as  a  result  of  two  being  tested 
in  the  same  room  at  the  same  time.  The  examinees  had  to  listen  closely 
to  the  administrator  t©  'understand  what  they  were  being  instructed  to  do. 
The  instructions  were  given  only  once.  No  instruction  was  repeated.  As¬ 
signment  of  examinees  to  administrators  was  random.  The  examinees  were 
seated  in  a  single  room.  The  administrators  drew  the  examinees  from  the 
front  row  and  worked  back  to  those  on  the  back  row0  As  each  examinee  was 
tested  he  was  sent  out  of  the  building  and  to  other  duty.  This  removed 
the  possibility  of  conference  between  men  who  had  been  tested  and  those 
who  had  not. 


Seven  persons  acted  as  behavior  test  administrators.  The  examinees 
were  tested,  the  same  day  with  the  behavior  test  and  Fora  A  of  the  MDAF 
F.nglish  Proficiency  Examination.  Foreign  Pre-Flight  students  from  eight 
countries  comprised,  the  107-case  sample.  The  countries  and  their  respective 
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numbers  ©f  cases  are  presented  in  Table  7  along  with  mean  scores  and  stand¬ 
ard  deviations  on  the  behavior  test# 


Table  7 

Distribution  Statistics  for  the  Behavior 
Test  Sample  by  Country 


Country 

N 

M 

SD 

Belgium 

19 

21.2 

4.3 

Cuba 

6 

18.2 

2.3 

Holland 

23 

26.1 

1.6 

Italy 

20 

21.8 

2.6 

Norway 

23 

24.4 

1.9 

Iran 

10 

20.4 

3.4 

Columbia 

5 

19.2 

4.7 

Nicaragua 

1 

28.0 

0.0 

Total 

107 

22.8 

3.8 

The  reliability  of  the  behavior  test  has  been  estimated  with  Kuder- 
Richardson  Formula  210  This  estimate  produced  a  coefficient  of  «70. 

While  moderate  in  size,  this  coefficient  is  acceptable  for  a  test  of  28 
items  which  are  collected  in  interview  fashion.  The  behavior  test  is 
deemed  to  have  sufficient  reliability  for  use  as  a  criterion. 

As  a  further  evaluation  of  the  behavior  test  procedure,  an  analysis 
of  administrator  differences  has  been  made®  The  examinees  were  assigned 
to  administrators  in  a  random  fashion.  While  there  were  seven  adminis¬ 
trators  in  all,  two  of  these  seven  administered  27  tests  each.  The  other 
five  administrators  each  gathered  fewer  cases.  To  test  the  hypothesis 
that  no  differences  existed  between  these  two  administrators,  an  analysis 
of  covariance  was  computed  as  follows.  The  cases  collected  by  adminis¬ 
trators  A  and  B  (27  cases  each)  were  considered  as  samples  A  and  B.  The 
means  of  the  two  samples  were  equated  on  the  basis  of  both  written  and 
oral  scores  from  the  MDAP  EPE.  Then  the  adjusted  means  of  the  behavior 
test  scores  in  the  two  samples  were  compared,  using  the  F  ratio.  This 
procedure  is  described  by  Johnson  (l).  No  significant  difference  was 
found  between  the  adjusted  behavior  test  means  for  the  two  samples. 

This  result  supports  the  hypothesis  that  no  differences  existed  between 
the  procedures  used  by  the  two  administrators.  Thus,  it  made  no  differ¬ 
ence  in  an  examinee’s  score  which  of  these  administrators  gave  him  the 
behavior  test.  A  summary  of  the  covariance  analysis  is  presented  in 
Table  8. 
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Summary  of  Covariance  Analysis  on  Mean  Behavior  Test  Scores 
Of  Groups  Tested  by  Different  Administrators 

(Sample:  54  MDAP  Pre-Flight  students) 


Adjusted 


Variation 

ii 

ssy 

ssv 

xi 

ss„ 

x2 

SPyxj 

SP 

y*2 

SF\  V 

xlx2 

iL 

SSy 

MSy 

Within 

52 

826.82 

6657.33 

890.67 

1131.33 

490.89 

1474.67 

50 

532.19 

10.64 

Between 

1 

4.17 

8.17 

10.67 

-5.83 

-6.67 

9.33 

1 

13,13 

13.13 

Total 

53 

830.98 

6665.50 

901.33 

1125.50 

481.22 

1484.00 

51 

545.32 

Note.~y:  Behavior  test  scores. 

Xj:  Scores  on  Part  I  (written)  of  MDAP  English  Proficiency  Examination. 
*2:  Scores  on  Part  II  (oral)  of  MDAP  English  Proficiency  Examination. 
ap  >  .05 


Having  developed  an  independent  measure  of  English  comprehension 
ability  with  satisfactory  reliability  and  collected  a  107-case  sample  in 
which  both  behavior  test  scores  and  EPE  scores  were  available,  the  re¬ 
lationship  between  the  measures  was  determined.  Correlation  coefficients 
were  computed  between  written  and  oral  scores  from  the  MDAP  EPE  Form  A 
and  the  behavior  test  scores.  The  results  are  presented  in  Table  9* 

The  correlation  coefficients  between  the  MDAP  EPE  part-scores  and 
the  scores  from  the  behavior  test  are  considered  as  estimates  of  how  well 
the  proficiency  examination  measures  ability  to  comprehend  English.  In 
this  light,  they  are  "validity  coefficients."  Thus,  the  coefficients  of 
.60  for  the  written  score  and  .65  for  the  oral  score  show  the  MDAP  EPE 
Form  A  to  have  construct  validity  as  a  measure  of  English  comprehension 
ability. 

The  coefficients  of  *60  and  .65  are  more  impressive  when  considered 
in  the  light  of  the  estimated  reliability  of  the  two  instruments.  If  the 
tests  are,  or  could  be  made,  more  reliable,  the  proficiency  test  scores 
might  produce  higher  coefficients  when  correlated  with  behavior  test 
scores.  The  coefficients  estimating  the  relationships  between  proficiency 
test  scores  and  behavior  test  scores  are  limited  in  size  by  the  imperfect 
reliability  of  both  instruments. 


English  Language — Proficiency  Examination  No.  I 

The  MDAP  EPE  Form  A  was  compared  with  another  test  independently  de¬ 
veloped  by  instructors  in  the  Pre-Flight  school,  English  Language — Profi¬ 
ciency  Examination  No.  I.  The  Pre-Flight  test  was  composed  of  20  items 
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Table  9 


Intercorrelations  and  Distribution  Statistics 
For  MDAP  English  Proficiency  Examination 
Part-Scores  and  Behavior  Test  Scores 

(Sample:  107  MDAP  Pre-Flight  students) 


Variable 

1 

_2_  JL 

M 

SD 

1. 

Part  I  (written) 

.68  .60 

72.0 

10.8 

2. 

Part  II  (oral) 

.68 

.65 

17.6 

4.1 

3. 

Behavior  test 

.60 

065 

22.8 

3.8 

presented  orally  and  130  items  presented  in  written  form.  The  written 
items  were  comparable  in  subject  matter  to  those  in  MDAP  EPE  Form  A0  The 
Pre-Flight  test  was  used  to  place  MDAP  students  in  three  classes  of  ability 
for  English  training.  Scores  on  MDAP  EPE  Form  A  and  the  Pre-Flight  place¬ 
ment  test  were  collected  for  203  foreign  students  in  Pre-Flight  training. 
These  scores  were  then  correlated  as  shown  in  Table  10. 


Table  10 

Distribution  Statistics  and  Correlation  Coefficients 
For  the  MDAP  English  Proficiency  Examination  Form  A 
And  a  Pre-Flight  Placement  Test 

(Sample:  203  foreign  student  pilots) 


Test 

Mean 

SD 

1. 

Part  I  (written) 

73.08 

9.10 

2. 

Part  II  (oral) 

17.72 

3.65 

3. 

Pre-Flight  placement  test 

73.37 

15.12 

The  fact  that  the  two  measuring  instruments  were  developed  independent¬ 
ly  to  measure  the  same  function  makes  the  correlations  important.  The  co¬ 
efficients  of  .78  and  .66  show  the  two  tests  to  be  much  alike0  This  result 
lends  confidence  in  the  use  of  both  tests* 
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Fyedlctirf  Validity 


Haring  determined  that  tbs  M94P  EPE  scores  are  reasonably  valid  esti¬ 
mates  of  ability  to  comprehend  English,  attention  turned  to  the  relation¬ 
ships  between  these  scores  and  success  in  Pre-Flight  school*  The  results 
from  Pre-Flight  school  vers  chosen  for  investigation  because  they  are  the 
first  school  success  info-mstion  to  become  available#  Upon  completion  of 
the  training  period  the  status  of  each  student  was  determined  by  one  of 
three  categories k  graduate,  eliminee,  or  holdover# 

A  376-  case  sample  cf  MDAP  Pre-Flight  students  was  collected ®  Each 
student  had  been  tested  overseas  with  the  MDAP  EPE  Fora  A  and  selected,  on 
the  basis  of  English  proficiency*  to  participate  in  USAF  flying  trainingo 
Further,  each  case  had  been  classified  at  the  conclusion  of  Pre- Flight 
training  as  graduate,  elimir.es,  or  holdover#  Table  11  shows  the  number  of 
cases  by  country  in  each  of  the  three  classifications#  The  mean  scores  for 
the  several  groupings,  are  presented  in  Table  12® 

Table  11 

Number  of  Cases  in  the  Disposition 
Categories  by  Country 


Country 

Graduate 

Eliminee 

Holdover 

Total 

Franco 

219 

3 

32 

254 

Holland 

16 

0 

0 

16 

Belgium 

38 

1 

1 

40 

Denmark 

21 

0 

0 

21 

Turkey 

19 

4 

15 

38 

Portugal 

7 

0 

0 

7 

Total 

320 

8 

48 

376 

Other  than 

French 

103 

5 

16 

122 

To  determine  the  relation ships  between  test  scores  from  the  MDAP  EPE 
and  success  in  Pre-Flight  training,  biserial  correlation  coefficients  were 
computed  using  graduation  as  the  upper  category  of  the  dichotomy,  and 
elimination  pluo  holdover  ao  the  lower  category®  The  coefficients  and  dis¬ 
tribution  statistics  are  prerented  in  Table  13# 

The  correlation,  coefficients  in.  Table  13  show  that  a  definite  positive 
relationship  exists  between  both  written  and  oral  part-scores  of  the 
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Table  12 


Mean  Written  and  Oral  Scores  for  Two  Groups 
Of  MDAP  Pre-Flight  Students 
By  Disposition  Groups 


Graduate 

Eliminee 

Holdover 

Group 

N 

Mw 

Mo 

N 

Mw 

Mo 

N_ 

Mw 

Mo 

French 

219 

62.8 

13  <3 

3 

54.0 

12.0 

32 

52.5 

11.6 

Other  than 
French 

101 

70ol 

16.7 

5 

64.O 

11.0 

16 

64.1 

11.8 

MDAP  EPE  and  success  in  Pre-Flight  training  among  foreign  students.  It 
should  be  noted  that  while  the  graduates  form  by  far  the  largest  percentage 
of  the  total  group  in  the  sample ,  all  correlation  coefficients  are  signifi¬ 
cant  beyond  the  .01  level  of  confidence. 


Table  13 

Distribution  Statistics  and  Correlation  Coefficients  Between 
Written  and  Oral  English  Proficiency  Scores  and  Pass/Fail 
And  Holdover  in  Pre-Flight  Training  Among  MDAP  Students 


Nt 

Mu 

Ml 

Mt 

SD  t 

P 

rbis 

®®rbis 

Part  I 

(written) 

French 

254 

62.89 

52.51 

61.46 

9.14 

.862 

.61 

.075 

Other 

122 

70oll 

64.14 

69.08 

7.43 

.828 

.45 

.116 

Total 

376 

65.20 

56088 

63.96 

9.36 

.851 

.49 

S 

0 

0 

Part  II 

(oral) 

French 

254 

13.36 

11.59 

13.11 

3.45 

.862 

.28 

.093 

Other 

122 

16.74 

11.64 

15.86 

4.35 

.828 

.66 

.095 

Total 

376 

14.42 

11.61 

14.00 

3.98 

.851 

.39 

.071 

The  present  data  show  the  written-part  scores  to  be  more  highly  re¬ 
lated  to  school  success  among  French  students  than  among  foreign  students 
from  countries  other  than  France.  On  the  other  hand,  the  oral  scores 
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are  more  highly  related  to  school  success  among  foreign  students  from 
countries  other  than  France  than  among  French  students*  Information  is  not 
available  in  the  present  data  to  explain  why  these  differences  occurred. 


SUMMARY  AND  CONCLUSIONS 


This  report  has  recounted  the  procedures  used  in  development  of  MDAP 
English  Proficiency  Examination  (EPE)Form  A.  Reliability  of  the  test  has 
been  estimated.  Distribution  statistics  for  several  samples  of  foreign 
military  personnel,  as  well  as  one  of  basic  airmen,  have  been  reported. 
Validity  has  been  considered  from  two  points  of  view,  construct  validity 
and  predictive  validity.  The  major  conclusions  drawn  from  the  above  in¬ 
formation  are  these: 

1.  The  MDAP  (F.PE) Form  A  is  a  measure  of  ability  among  foreign 
student  pilots  to  comprehend  both  written  and  oral  English. 

2.  The  test  has  predictive  validity  for  Pre-Flight  training  and 
is  acceptably  reliable  for  use  as  a  selective  instrument. 

3.  Both  written  and  oral  parts  have  suitable  reliability  and 
validity  and  are  only  moderately  related  to  each  other.  They  can  effec¬ 
tively  be  used  separately  as  S3 lection  devices;  a  person  qualifies  on 
both  parts  of  the  teat  to  become  eligible  for  entry  into  pilot  training. 

Future  research  involving  the  MDAP  EPE  Form  A  should  include  investi¬ 
gation  of  differences  between  item  types  as  to  their  effectiveness  in  meas¬ 
uring  English  comprehension.  Teat  scores  should  also  be  related  to  success 
in  Primary  and  Basic  Pilot  Training. 
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